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Abstract: Three dimensional quantification of organ shape and structure 
during minimally invasive surgery (MIS) could enhance precision by 
allowing the registration of multi-modal or pre-operative image data 
(US/MRI/CT) with the live optical image. Structured illumination is one 
technique to obtain 3D information through the projection of a known 
pattern onto the tissue, although currently these systems tend to be used 
only for macroscopic imaging or open procedures rather than in endoscopy. 
To account for occlusions, where a projected feature may be hidden from 
view and/or confused with a neighboring point, a flexible multispectral 
structured illumination probe has been developed that labels each projected 
point with a specific wavelength using a supercontinuum laser. When 
imaged by a standard endoscope camera they can then be segmented using 
their RGB values, and their 3D coordinates calculated after camera 
calibration. The probe itself is sufficiently small (1.7 mm diameter) to allow 
it to be used in the biopsy channel of commonly used medical endoscopes. 
Surgical robots could therefore also employ this technology to solve 
navigation and visualization problems in MIS, and help to develop 
advanced surgical procedures such as natural orifice translumenal 
endoscopic surgery. 

© 201 1 Optical Society of America 

OCIS codes: (330.1710) Color, measurement; (110.6880) Three-dimensional image acquisition; 
(170.1610) Clinical applications; (170.2150) Endoscopic imaging; (170.3890) Medical optics 
instrumentation 



References and links 

1. D. Stoyanov, G. P. Mylonas, F. Deligianni, A. Darzi, and G.-Z. Yang, "Soft-tissue motion tracking and structure 
estimation for robotic assisted MIS procedures," in Medical Image Computing and Computer- Assisted 
Intervention— MICC AI 2005, J. Duncan, and G. Gerig, eds. (Springer- Verlag, 2005), pp. 139-146. 

2. S. Voros, J.-A. Long, and P. Cinquin, "Automatic detection of instruments in laparoscopic images: a first step 
towards high-level command of robotic endoscopic holders," Int. J. Robot. Res. 26(1 1-12), 1 173-1 190 (2007). 

3. J. J. Schwartz and G. R. Lichtenstein, "Magnification endoscopy, chromoendoscopy and other novel techniques 
in evaluation of patients with IBD," Tech. Gastrointest. Endosc. 6(4), 182-188 (2004). 

4. S. Kato, K. I. Fu, Y. Sano, T. Fujii, Y. Saito, T. Matsuda, I. Koba, S. Yoshida, and T. Fujimori, "Magnifying 
colonoscopy as a non-biopsy technique for differential diagnosis of non-neoplastic and neoplastic lesions," 
World J. Gastroenterol. 12(9), 1416-1420 (2006). 

5. P. J. Edwards, A. P. King, D. J. Hawkes, O. Fleig, C. R. J. Maurer, Jr., D. L. Hill, M. R. Fenlon, D. A. de Cunha, 
R. P. Gaston, S. Chandra, J. Mannss, A. J. Strong, M. J. Gleeson, and T. C. Cox, "Stereo augmented reality in the 
surgical microscope," Stud. Health Technol. Inform. 62, 102-108 (1999). 

6. P. Pratt and D. Stoyanov, Visentini-Scarzanella, and G.-Z. Yang, "Dynamic guidance for robotic surgery using 
image-constrained biomechanical models," in Medical Image Computing and Computer- Assisted Intervention — 
MICCAI2010, T. Jiang, N. Navab, J. P. W. Pluim, and M. A. Viergever, eds. (Springer- Verlag, 2010), pp, 77- 
85. 

7. G. P. Penney, D. C. Barratt, C. S. K. Chan, M. Slomczykowski, T. J. Carter, P. J. Edwards, and D. J. Hawkes, 
"Cadaver validation of intensity-based ultrasound to CT registration," Med. Image Anal. 10(3), 385-395 (2006). 



#152969 - $15.00 USD Received 16 Aug 201 1; revised 19 Oct 201 1; accepted 21 Oct 201 1; published 25 Oct 201 1 
(C) 201 1 OS A 1 November 201 1 / Vol. 2, No. 1 1 / BIOMEDICAL OPTICS EXPRESS 3119 



8. C. J. Moore, P. J. Sharrock, F. Lilley, and D. Burton, "3D body surface measurement and display in radiotherapy 
part III: respiration and deformation in post-surgical breast cancer patients," in International Conference on 
Medical Information Visualisation — BioMedical Visualisation (MedVis '06) (IEEE, 2006), pp. 109-114. 

9. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision (Cambridge University Press, 2000). 

10. D. Stoyanov, A. Darzi, and G.-Z. Yang, "Dense 3D depth recovery for soft tissue deformation during robotically 
assisted laparoscopic surgery," in Medical Image Computing and Computer-Assisted Intervention — MICCAI 
2004, C. Barillot, D. R. Haynor, and P. Hellier, eds. (Springer- Verlag, 2004), 41^8. 

11. D. Stoyanov, A. Darzi, and G.-Z. Yang, "A practical approach towards accurate dense 3D depth recovery for 
robotic laparoscopic surgery," Comput. Aided Surg. 10(4), 199-208 (2005). 

12. J. Geng, "DLP -based structured light 3D imaging technologies and applications," Proc. SPEE 7923, 79320B, 
79320B-15(2011). 

13. C. J. Moore, D. R. Burton, O. Skydan, P. J. Sharrock, and M. Lalor, "3D body surface measurement and display 
in radiotherapy part I: technology of structured light surface sensing," in International Conference on Medical 
Information Visualisation — Biomedical Visualisation (MedVis '06) (IEEE, 2006), pp. 97-102. 

14. T. T. Wu, T.-H. Cheung, S.-F. Yim, and J. Y. Qu, "Optical imaging of cervical precancerous lesions based on 
active stereo vision and motion tracking," Opt. Express 16(15), 1 1224-1 1230 (2008). 

15. M. Chan, W. Lin, C. Zhou, and J. Y. Qu, "Miniaturized three-dimensional endoscopic imaging system based on 
active stereovision," Appl. Opt. 42(10), 1888-1898 (2003). 

16. J. Batlle, E. Mouaddib, and J. Salvi, "Recent progress in coded structured light as a technique to solve the 
correspondence problem: a survey," Pattern Recognit. 31(7), 963-982 (1998). 

17. J. Salvi, J. Pages, and J. Batlle, "Pattern codification strategies in structured light systems," Pattern Recognit. 
37(4), 827-849 (2004). 

18. H. J. Chen, J. Zhang, and J. Fang, "Surface height retrieval based on fringe shifting of color-encoded structured 
light pattern," Opt. Lett. 33(16), 1801-1803 (2008). 

19. T. Smith and J. Guild, "The CLE. colorimetric standards and their use," Trans. Opt. Soc. 33(3), 73-134 (1932). 

20. V. C. Smith and J. Pokorny, "Color matching and color discrimination," in The Science of Color, S. K. Shevell, 
ed. (Elsevier, 2003). 

21. P. Kazanzides, S. DiMaio, A. Deguet, B. Vagvolgyi, M. Balicki, C. Schneider, R. Kumar, A. Jog, B. Itkowitz, C. 
Hasser, and R. Taylor, "The surgical assistant workstation in minimally invasive surgery and microsurgery," in 
MICCAI Workshop on Systems and Arch, for Computer Assisted Interventions (2010), 

http://hdl.handle.net/10380/3179. 



1. Introduction 

Computer vision techniques have recently been applied in minimally invasive surgery (MIS) 
to provide more information to the surgeon from the white light endoscopic view and to 
compensate for the natural cues that are sacrificed when moving from the open surgery 
platform, for instance the loss of haptic feedback or stereo vision. These techniques may be 
used to track the surgical instruments, predict the motion of the tissue or detect the tissue 
surface profile [1,2], but they also have the potential to improve surgical accuracy and 
reliability by providing augmented views of the tissue and enabling diagnostic assistance. 

The detection of tissue surface information in three dimensions has many possible 
applications in MIS including the registration of multimodal images or pre-operative image 
data, the improved navigation of robotic instruments and the measurement or characterization 
of tissue morphology for diagnostic purposes [3,4]. It is commonplace for preoperative MRI, 
CT or ultrasound scans of a patient to be used for surgical planning to locate and navigate to 
paronychial tumors. However during the surgery itself surgeons often rely on the optical 
identification of anatomical landmarks and navigate based on the relative positions between 
the static pre-operative images and the intraoperative view to make a judgment on how to 
dissect the tissue. A longstanding aim of the field of computer aided surgery is to help guide 
the surgery using an augmented reality view of the tissue revealing hidden structures that are 
present in the preoperative images but not in the live endoscopic view [5]. This could also 
allow the current finite element tissue models to be fitted to the preoperative data to allow the 
augmented view to deform and shift as the tissue is manipulated under known boundary 
constraints [6]. An example of the use of tissue surface morphology is the evaluation of the 
shape of colonic polyps. These have been shown to be related to the status of the lesion itself 
and have been proposed as a method of guiding or even replacing biopsies [3,4]. The accuracy 
of this type of technique is still down to the experience and judgment of the observer. 
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Quantification of polyp surface shape could be used as an objective measure of these polyps 
and compared against a library of known cases to guide biopsy selection. 

Due to the reasons mentioned above, it is a widely pursued goal of the surgical imaging 
community to obtain an intraoperative measurement of tissue surface geometry. This has been 
previously attempted to allow multimodal registration in cadaver measurements [7], 
microscopic imaging [5] and radiotherapy guidance [8], although MIS applications remain 
relatively unexplored. Arguably the most successful method of 3D surface measurement 
during MIS has been stereo reconstruction, where a rigid endoscope with two imaging 
channels simultaneously acquires corresponding 'left' and 'right' views of the scene and uses 
a mathematical model based on a pinhole camera assumption to triangulate the position of 
salient features (areas of high contrast, such as the intersection of two blood vessels) found in 
both channels [9-11]. However, this technique is not suited to homogenous or featureless 
tissues. In this case, structured lighting must be used to project artificial features onto the 
tissue surface. A number of techniques for extracting depth information using patterned light 
exist and involve projection of lines, grids or dots [12]. However, the brightness required and 
the size of the optics needed to provide the pattern has, thus far, mostly limited their use to 
biomedical applications outside of the endoscopic regime due to the difficulties in 
miniaturizing projective imaging systems [13,14]. Previous attempts at endoscopic 
implementation have only been suitable for rigid endoscopy and were not compatible with 
pattern encoding techniques which are more immune to tissue occlusions [15]. 

To achieve dense surface reconstruction, a number of projected features is required, which 
introduces computational problems in the identification of specific features in the presence of 
occlusions where parts of the tissue may obscure part of the pattern. This results in 
discontinuities and errors in the 3D reconstruction. To overcome this, the pattern must be 
coded so that each projected feature can be identified, thereby minimizing the risk that a 
feature is confused with a 'missing' one and causing an error in the reconstruction. In other 
applications, this has been done spatially (using binary words in the pattern for example) or by 
introducing a number of different colors into the pattern using digital projection technology 
[16-18]. However, these techniques were applied to non-endoscopic and in most cases non- 
biological situations. 

In this paper, an endoscopic fiber-optic structured lighting probe for intraoperative 3D 
imaging is proposed to overcome the limitations of the technologies described above. It uses a 
broadband laser to encode a dense pattern of spots with a unique color for each, and a 
segmentation algorithm that recovers the spectral signature of each dot from standard RGB 
color CCD images. The probe is highly flexible, has a small diameter and produces a pattern 
of high brightness making it suitable for use with biological tissue and low light collection 
efficiency endoscope systems. The optical set-up and spot identification algorithm are 
described, along with initial results characterizing the performance and limitations of the 
system and 3D reconstructions of ex vivo tissue. 

2. Materials and methods 

2.1. Probe design 

The optical set-up in Fig. 1 shows how the spectrally encoded pattern is generated. The 
collimated output of a 4 W supercontinuum laser (SC400-4, Fianium Ltd., Southampton, UK) 
is dispersed using a prism (SF-1 1 glass) and focused into a thin line to couple it into the 127 
fibers of the array end of a custom-made line-to-spot converter (FiberTech Optica, Inc., 
Canada). The distances between the prism and the coupling lens, and the coupling lens and the 
fiber array are set to the lens focal length so that the spectral components are sharply focused 
at the linear array. At the distal end of the probe, a short focal length aspheric lens (f= 4.51 
mm) is used to form an image of the end face of the probe. Since the fibers are incoherently 
bundled, the result is that a random pattern of colored dots is projected onto the screen or 
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sample. The brass ferrule at the array end of the probe is 12.5 mm in diameter, but the rest of 
the probe has a maximum outer diameter of 1 .7 mm making it compatible with biopsy ports of 
commonly used flexible endoscopes. 




X I nm 

Fig. 1. (a) Broadband laser light from the supercontinuum is dispersed by an SF-11 prism, 
which is then coupled into the fibers (50 |im core) at the array end of the probe. The projected 
pattern is a magnified image of the end face of the bundle formed by the projection lens, (b) 
Emission spectrum of the supercontinuum laser source, with the wavelength range used by the 
probe indicated between the dashed lines. 

Since the light source is a broadband laser and the fibers are closely spaced at the input 
end (average separation of 68 ± 3 um), each projected spot effectively contains a unique low 
bandwidth spectral feature. It should be noted that although the laser is nominally 4 W, it 
emits over a 420-2200 nm range meaning that most of that power is contained in the near 
infrared, which is filtered out prior to reaching the prism. Only visible wavelengths are 
coupled into the fiber bundle so that the final output power at the probe tip after coupling 
losses is approximately 110 mW, well below the tissue damage threshold. 

2.2. Wavelength segmentation and centroiding 

The identification and segmentation is based on recovery of the peak wavelength of each spot 
using knowledge of the relationship between the RGB colorspace and the wavelength of the 
light. A schematic is shown in Fig. 2 to illustrate the transformation between RGB and 
wavelength for each pixel and is further described in the text below. 

The CIE 1931 xy colorspace is used to represent the colors of visible light in an area 
defined by wavelengths visible to the human eye [19]. The xy chromaticity coordinates of 
pure wavelengths form a curve called the spectrum locus at the boundary of the colorspace. 
The RGB values as recorded by a standard color digital camera however, are only capable of 
detecting a triangular subset of these colors defined by the RGB response of the camera 
indicated in Fig. 2 (b). When an RGB image of the spot pattern is recorded by a camera, the 
position of each pixel within this triangular space can be calculated with knowledge of the 
RGB system used. This is achieved for each pixel by converting the RGB triplet into the 
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Fig. 2. Spot segmentation and 3D calibration, (a) Cartoon image showing three projected spots, 
having different RGB values, (b) Each RGB triplet is converted to xy coordinates. A line 
projected through these coordinates from a reference white spot intersects the spectrum locus 
(dashed) at the dominant wavelength of the pixel, (c) RGB pixels are replaced by the calculated 
wavelength to form a greyscale 'i-space' image which can be thresholded to find the centroids 
of spots of a particular wavelength, (d) Epipolar geometry showing different positions of a 
calibration object (ci-c 3 ) and triangulation of points using spot centroids. 

tristimulus values XYZ, and then into the chromaticity coordinates xy by multiplication by the 
3x3 transformation matrix M. The elements of M are calculated based on the color co- 
ordinates of the vertices of the RGB triangle and the reference white [20]. 

The dominant wavelength detected at the pixel of interest is found by projecting a line 
from a reference white point, through the pixel's xy coordinates, to the point of intersection 
with the spectrum locus. In practice this is achieved computationally by creating a look-up 
table of color coordinates representing the mapping of the spectrum locus onto the triangular 
RGB gamut. For each pixel in an image, the intersection point of a line defined by the 
reference white and pixel xy, and the nearest side of the RGB gamut is calculated. The closest 
point to this intersection is then located in the look-up table and its corresponding wavelength 
returned (Fig. 2 (b)). The result is a greyscale '/l-space' image where each pixel value is the 
calculated wavelength of the light detected. 

Once the 'A-image' is acquired, spots of a particular wavelength can be isolated by 
thresholding by wavelength and finding their centroids. The finite bandwidth of each 
projected spot, sensor noise and uncertainty in locating an RGB triplet's dominant wavelength 
means that each spot consists of a narrow distribution of wavelength values in A-space. 
Therefore a simple thresholding of the A-image at a wavelength of interest results in a cluster 
of pixels scattered within the region of the spot rather than a continuous area. A region- 
growing algorithm searches for these clusters by examining the nearest neighbors of each 
pixel in the thresholded image, and those with values within ± 1 nm of the wavelength of 
interest are deemed suitable for inclusion in the region. In this way, clusters of pixels at the 
location of a projected spot are 'grown' into each other to form a single identifiable region. As 
a final step, a median filter is applied to remove any erroneously detected pixels outside of the 
spot. The centroid of the detected region is then recorded along with the histogram of 
wavelength values within it. The peak value of this distribution is recorded as the 'label' for 
that particular spot and the process is repeated for multiple spots within the image. 

2.3. 3D reconstruction 

Triangulation of the 3D location of the imaged spots is accomplished through a calibration 
based on the pinhole camera model and methods previously developed for stereo 
reconstruction [9]. A calibration object (a plane with a checkerboard pattern of known 
dimensions) was imaged at 12 different positions in the camera's field of view. White light 
and patterned illumination images were recorded for each position of the object to acquire 
calibration data for calculating the intrinsic geometrical parameters of the camera and also for 
estimating the 3D path of light rays projected from the probe. 

Using the intrinsic results, it was possible to estimate the 3D position of the calibration 
object in metric space, so with the knowledge that the corresponding images of patterned 
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illumination must show the spots lying on these planes, their 3D locations could be calculated. 
For each calibration image of the projected spots, the centroid of each spot was located using 
the algorithm described in Section 2.2. Backprojecting each spot's rays to find the intersection 
with the calibration plane provided a set of 3D positions that the structured light ray passed 
through. Lines representing light rays were fitted to each spot's 3D data, composed of 12 
calibration positions, giving the origin of the projection and its position relative to the camera 
(Fig. 2 (d)). With the projection path of the rays mapped out, the 3D position of an individual 
spot reflected from an object of unknown shape could be determined by finding its position on 
the camera's image plane and projecting a ray through the 'pinhole' to intersect with its 
known projection path. Due to noise in the image the intersection is not guaranteed and the 
solution is taken as the midpoint of the shortest line connecting the two rays [9]. 

2.4. Characterization and ex vivo testing 

To test the effectiveness of the algorithm in recovering the wavelength of each spot, the 
pattern was projected onto a white screen and RGB images were recorded using a color CCD 
camera (DCU 223 C, Thorlabs Ltd., UK). The white screen was then removed and spectra of 
individual spots were recorded using an optical fiber probe placed in the pattern's plane and 
connected to a spectrometer (HR 4000, Ocean Optics, Inc., USA). 

Since the optical properties of tissue within the body may vary significantly, the effect of 
background reflectivity on the detected spot wavelength was examined by imaging the pattern 
on different colored backgrounds. The performance of the system was then evaluated in 
biological tissue of varying optical properties using ex vivo kidney (ovine) and intestinal 
(porcine) tissue. Finally, the 3D reconstruction capability of the system was demonstrated by 
calibrating the projection probe and camera, and determining the shape of two validation 
objects (plane and cylinder) as well as ex vivo tissue of varying type, color and shape (kidney, 
fat and liver). 

3. Results 

3.1. Characterization: algorithm testing 

Figure 3 (a) shows the raw RGB image acquired by the camera when the pattern was incident 
on a plane white screen, while Fig. 3 (b) shows the same pattern after conversion to A-space. 




x (measured) / nm 

Fig. 3. (a) RGB image of pattern recorded by camera, (b) 1-space image with centroids of 
spots, (c) Plot showing spot wavelength as calculated by the segmentation algorithm against the 
wavelength measured using a spectrometer. The transmission response of the camera's filters 
(normalized to 1) is overlaid and the identity line is shown in black. The error bars indicate ± 1 
standard deviation. 
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The wavelengths of the spots in the A-space image after processing are plotted against the 
wavelengths measured with the spectrometer in Fig. 3 (c). In order to distinguish closely 
spaced wavelengths reliably, the algorithm requires a signal in at least two out of the three 
channels (red, green and blue), which define the working space of the camera's RGB system. 
The filter response of the camera used, as measured in the lab using a reflectance standard and 
tunable filter, is also plotted in Fig. 3 (c) and shows that the areas of the plot with the strongest 
correlation between calculated and measured wavelength correspond to spectral regions with 
strong overlap between adjacent channels. 

The wavelength plot is marked by two areas of strong correlation separated by flat regions 
in the red (X > 600 nm) and green (530 < X < 560 nm), with significant noise in the blue (X < 
490 nm) due to low signal levels. 

Figure 4 shows how the detected spot wavelength varies with background optical 
properties. When card colored at opposite ends of the spectrum (red and blue) was used as a 
background (Fig. 4 (a)), a strong correlation between the calculated wavelength of 
corresponding spots is seen and all values lie along the identity line (with the exception of a 
number of points at the far end of the red region; X > 640 nm). 




Fig. 4. Calculated wavelength of a set of spots projected onto surfaces of different colors, (a) 
Blue and red card, (b) Ex vivo tissue: porcine intestine (inset, top left) and lamb kidney (inset, 
bottom right). The error bars represent ± 1 standard deviation. 

For ex vivo tissue (Fig. 4 (b)), the wavelength values are again scattered along the identity 
line showing that the values outputted by the algorithm are in broad agreement in the 450-500 
nm and 550-350 nm regions despite the noticeable difference in tissue color (inset). There is a 
noticeable increase in error however, at the blue end of the spectrum where a number of points 
deviate from the expected values due to the lower signal levels. 

3.2. 3D tissue surface reconstruction 

The results of the projector-camera calibration are shown in Fig. 5 (a) and indicate the 
relationship between the origin and propagation direction of the 'best fit' rays emitted from 
the probe tip and their intersection with the calibration object at different positions. The 
average error in fitting each 'ray' to its set of calibration points was 0.32 mm. 

Two opaque validation objects were used to test the system's 3D reconstruction. These 
were a flat object (black planar plastic divider, thickness 2 mm) and a brown polymer cylinder 
(radius 40.5 mm), whose 3D coordinates are shown in Figs. 5 (b) and (c) respectively. A mesh 
surface was fitted to the 3D point clouds using a least-squares minimization algorithm 
implemented in Matlab (The MathWorks, Inc., USA). Reprojection errors using the calibrated 
rays were 1.3 pixels in x and 1.06 pixels in y. The data in Fig. 5 (b) were fit by a planar 
surface with an error of 0.05 mm. The reconstructed cylindrical object was fitted by a cylinder 
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Fig. 5. Three dimensional calibration and validation, (a) Origin and propagation direction of 
projected spots with respect to the camera (origin) as determined during calibration routine, (b) 
Planar object, (c) Cylindrical object, diameter =81 mm. (d) Cross-section of cylindrical object 
with least-squares fit. 




Fig. 6. Three-dimensional reconstruction of ex vivo tissue, (a) Porcine liver, 'step', (b) Ovine 
kidney, convex curve, (c) Porcine liver, convex curve, (d) Porcine tissue, border between fatty 
tissue and liver, (e) Porcine liver, 'valley'. 

with radius matching that of the validation object (40.5 mm) to sub-millimeter accuracy (rms 
difference = 0.15 mm) as shown in Figs. 5 (c) and (d). 

Following the validation experiments, a number of samples of ex vivo tissue were 
examined to demonstrate operation of the probe and analysis algorithms over varying 
background optical properties and curvatures. The results in Fig. 6 show reconstructed 3D 
data for ovine kidney, porcine liver and fatty tissue, and have varying physical features 
(convex/concave curve, 'step' discontinuity and miscellaneous 'fine' structure). The 
reconstructed surface profiles were observed to match well the observed surface profiles of 
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the tissues, which can be observed in the color photographs presented. These tests 
demonstrate that the technique may be applied on ex vivo tissue, while the validation with the 
test objects demonstrates the accuracy that may be obtained. 

4. Discussion 

The structured lighting probe described in this paper seeks to address the requirements for 
pattern projection in minimally invasive surgery by providing a pattern of high brightness, 
high density and employing a codification strategy based on spectral discrimination. The 
dispersed light from the supercontinuum laser ensures that each projected feature is labeled 
with a unique color which can then be detected from the camera RGB image using the 
segmentation algorithm described. 

Initial results have shown that by using the segmentation algorithm with knowledge of the 
camera's RGB system it is possible to identify the wavelength of individual spots. However, 
the performance of this algorithm is limited to regions of the spectrum where there is an 
overlap between the transmission spectra of adjacent filters (blue/green, green/blue). Outside 
of these regions, the camera only perceives a red, green or blue spot of varying intensity 
resulting in a flat response below 490 nm, above 600 nm and between 530 and 560 nm. 

Using flat screens of different colors as objects, the calculated wavelength of individual 
spots was observed to be constant. This is a feature of this structured lighting probe due to the 
sharp spectra of the individual spots (FWHM «5 nm), which meant that the observed color did 
not change and only a variation in spot intensity was observed as the background reflectance 
spectrum was altered. In biological tissues the same color invariance is observed, particularly 
in the overlapping spectral regions identified earlier. However, there are noticeable errors in 
the blue region caused by the diffusion of longer wavelength light from neighboring spots 
through the tissue between neighboring spots. The intestinal tissue sample was not as strongly 
absorbing in the 400-500 nm region as the kidney, meaning that adjacent spots were more 
likely to be influenced by scattered light. For the blue spots this had the effect of biasing the 
calculated wavelength towards the green. The blue spots were more susceptible to this effect 
by green rather than red spots due to the relatively lower number of red spots in close 
proximity. Strong absorption by hemoglobin at wavelengths below 500 nm also led to low 
reflected intensities at these wavelengths. 

For non-biological validation objects, reconstruction of their 3D shape in metric space was 
demonstrated. High accuracy was achieved, with reprojection errors on the order of one pixel 
in the calibration and sub-millimeter accuracy in measuring a cylindrical and planar validation 
objects. 

Sections of ex vivo tissue were also reconstructed and visually appeared to match the 
tissue surfaces observed. For each tissue type the morphology could be distinguished, 
including the convex curves, valley and 'step change'. However, fine structure in the fatty 
tissue was not resolved due to undersampling of the surface by the patterned light at that 
working distance, where the inter-spot spacing was of the order of several millimeters. This 
means that at long working distances this probe is limited to resolving sparse 3D structure 
with a resolution of less than a centimeter. However, at short working distances (2 cm or less) 
the pattern diameter could be reduced to «2 cm and the inter-spot separation to «2 mm or less, 
which would make more dense reconstruction possible. This would also be compatible with 
potential applications such as quantification of polyp shape during colonoscopy, where typical 
polyps can have diameters between 7 and 10 mm [4]. The segmentation and centroiding 
algorithm worked well in the green-red region of the spectrum, but a number of spots at the 
blue end could not be segmented and used in the reconstruction, reducing the accuracy of the 
final surface fit. 

More calibration images could improve the accuracy of the calibration and hence the 
accuracy of reconstruction, however, the main source of error in the 3D reconstruction is in 
the detection of the spots. Their shape is dependent on distance and surface orientation, and 
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any errors, from a partial occlusion of a spot for example, means that the centroid does not 
reflect the correct ray path. The amount by which these image measurement errors are 
converted into triangulation errors is determined by the 'baseline' (distance between the 
projector and camera). Previous results from stereo endoscopy have already demonstrated that 
reliable reconstructions can be achieved using a small baseline («5 mm) if the working 
distance is sufficiently short (< 5 cm) [1 1,21], which suggests that a future application for this 
device could be a colonoscope. 

5. Conclusions 

A structured lighting probe has been built that is capable of delivering patterned light to the 
tissue using a compact fiber probe that is compatible with existing endoscopic devices. The 
optical set-up addresses the correspondence problem by coding the pattern with different 
wavelengths, assigning a unique wavelength to each projected spot. This can minimize the 
problem of occlusion, common to structured lighting set-ups, by tracking exactly which 
projected features have been blocked from view and avoid the risk of detected spots being 
matched to the calibrated rays of 'missing' spots. 

A segmentation algorithm based on the CIE 1931 chromaticity diagram has been shown to 
successfully recover the wavelengths of individual spots in areas of the spectrum where there 
is an overlap between the transmission spectra of the camera's RGB filters. In measurements 
on colored paper and tissue of varying optical properties it was shown that the calculated 
wavelength did not change with background reflectivity in the optimum region. This means 
that it will be possible to use the probe on a variety of tissues or at organ boundaries where 
optical properties may vary significantly. 

Future work will be focused on optimizing the spectral output of the probe so that the 
number of spots with wavelengths in the optimal blue/green and green/red regions is 
maximized. A customized filter arrangement with increased spectral overlap will also be used 
to extend the range of these spectral regions. Spatial constraints will be added to the algorithm 
to aid discrimination of any spots that appear to have the same wavelength. To overcome 
some of the limitations imposed by absorption in tissue and to allow for practical clinical use, 
an interleaved imaging system will also be implemented. This would involve a high speed 
synchronized camera/shutter system to acquire patterned and white light images alternately in 
such a ratio as to minimize the visual impact of the pattern and make simultaneous normal 
white light viewing and 3D data acquisition possible. High-speed acquisition sequences of 
varying exposure times would also enable the acquisition of composite high dynamic range 
images where blue spots are made visible in the presence of hemoglobin without saturating 
those closer to the red end. Furthermore, absolute validations of the instrument accuracy in 
measurement of tissue will be carried out using ground truth data from CT images in order to 
better understand and minimize sources of error arising from diffusion and absorption of light. 
We believe that these modifications will allow a clinical structured lighting system to be 
constructed. 
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